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DETAILED ACTION 

Response to Arguments 

1 . Applicant's arguments filed 4/28/2010 have been fully considered but they are 
not persuasive. 

At page 9, the Applicant argues that Wu does not cure the alleged deficiencies of 
Wilinski and Zheng. The reason why the Applicant believes so is unclear, since the 
Applicant merely describes how the method of line integral in Wu is applied to a classic 
shape from shading algorithm by Horn. The Examiner can however venture that the 
Applicant intends that Wu does not teach the limitation because Wu integrates surface 
normals and further because it requires the designation of an arbitrary depth to a point 
in the image. First, the Applicant cannot merely argue the deficiencies in Wu as a way 
to overcome the rejection of the claims. The claims are rejected as obvious over 
Wilinski, Zheng and Wu, and the Applicant's argument does not address why the claims 
limitations are obvious in view of these references, but rather the absence of explicit 
recitation of the limitation in either of these references separately. Notoriously absent 
from the Applicant's argument is any reasoning of why it would not have been obvious 
to one of ordinary skill in the art at the time of the invention to apply line integrals such 
as those taught by Wu to the depth derivatives taught by Wilinski and Zheng. The 
Examiner does not adopt and has never adopted the position that the line integrals in 
Wu are equal to those claimed. What the Examiner considers obvious, however, is that, 
since Wilinski and Zheng provide information of depth as a change in depth across 
boundaries, integrating these differences is an obvious way of obtaining global depth 
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data. On the subject of why Wu requires an initial point to be assigned an arbitrary 
depth, the Examiner first notes that nothing in the claims precludes setting a point with 
arbitrary depth and then calculating the depth of all points relative to that point. Further, 
this feature of Wu is not included in the combination, and it is unclear whether the 
Applicant intends to argue that without it line integrals cannot be performed, or simply, 
and as above, that this is different from what is claimed. If the first, the Applicant 
provides no arguments that it is necessary. If the latter, as above, this is not relevant, 
since as indicated the feature is not used in the combination. 

The Examiner respectfully disagrees with the Applicant's positions, but 
nevertheless has added to the rejection the above mentioned reference in order to 
clarify the rejection, which is apparently not properly understood by the Applicant. 

2. Applicant's arguments with respect to claims 1 , 2, 4, 5, and 7-1 7 have been 
considered but are moot in view of the new ground(s) of rejection. 

Claim Rejections - 35 USC § 103 

3. The following is a quotation of 35 U.S.C. 1 03(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

4. Claims 1-4 and 6-13 and 15-17 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Wilinski et al. (WIPO Publication No. 02/095680; "Wilinski" 
hereinafter) in view of Zheng et al. (Q. Zheng, R. Chellappa; Estimation of llluminant 
Direction, Albedo, and Shape from Shading; IEEE Transactions of Pattern Analysis and 
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Machine Intelligence, Vol. 13, July 1991; "Zheng" hereinafter), Redert et al. (WIPO 
Publication No. 2004/066212; "AAPA" hereinafter) and Wu et al. (Z. Wu, L. Li; A Line- 
Integration Based Method for Depth Recovery from Surface Normals; IEEE, November 
1988; "Wu", hereinafter). 

Examiner Note: AAPA shares an assignee and an inventor with the instant 
application. The examiner cites portions in AAPA that describe the state of the art prior 
or at the time of the filing of AAPA. The Examiner considers these statements as 
Applicant admitted prior art, particularly since the filing date of AAPA antedates the 
instant application. 

Regarding claims 1, 15, 16 and 17, Wilinski teaches a method for generating 
from a single view input image, a depth map comprising depth values representing 
distances to a viewer, for respective pixels of the image (see p. 1, lines 10-18 for 
generating a depth map, see p. 2, lines 30-32 for a single image; see p. 8 for 
determining depth of pixels in the image by known methods) and assigning a depth 
value in a first group of depth values corresponding to the pixel (so that pixels belonging 
to the same image segment are assigned the same group of pixel values, see p. 6, lines 
1 and 2 for seed values composing a pixel; see p. 8, lines 31-34 and p. 9, lines 1 and 2 
for storing the depth values of the seed pixels; see p. 10, lines 26-28 for assigning to a 
pixel the depth value of the segment it belongs to). See p. 10, lines 21-30 for a system 
and units for performing the method, including an input of the digital image, and for 
preferably using a processor provided with a computer program to execute the method. 
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Wilinski does not explicitly teach computing cost values that comprise respective 
measures of a number of and extent of transitions in luminance and/or color 
components for pixels of the image on a path related to a spatial disposition of objects 
in the image, wherein said computing includes computing a cost value for a first one of 
the pixels of the image by accumulating differences between luminance and/or color 
values of pairs of neighboring connected pixels at transitions which are disposed on a 
path from the first one of the pixels to a second one of the pixels wherein the second 
one of the pixels belongs to a predetermined subset of the pixels of the image. Wilinski 
does teach detecting transitions between objects in the image by contours (see p. 7, 
lines 29 and 30 for segmentation based on luminosity; see p. 5, lines 1 2-1 8 for using a 
method for detecting edges by intensity changes); and as mentioned before assigning 
depth to pixels in those edges, and through them to pixels in their segment. Zheng 
however teaches a method to obtain shape from shading using contours (see abstract; 
see p. 684, 1 st col., 2 nd par.), where the value of the intensity difference across an edge 
is determined and if it is above a threshold, detects a relative depth variation between 
the segments (see p. 684, 2 nd col., 3 rd and 4 th pars.). Because Wilinski teaches 
performing segmentation of an image and depth determination of the pixels at the 
contours generated, and Zheng teaches determining the depth of edges in the image by 
examining intensity differences across edges, it would have been obvious to one of 
ordinary skill in the art at the time of the invention to combine the image segmentation 
and depth determination taught in Wilinski with the depth determination across edges 
taught by Zheng. This combination is implicitly suggested by Wilinski, which discloses 
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segmentation and depth determination of edge pixels by known methods, and expressly 
by Zheng, which discloses that the method can be improved by performing 
segmentation first; see the last three lines of page, 684, 2nd col., 3rd par. 

Neither Wilinski nor Zheng teach that the measurements for depth are taken 
along a path which results in a cost function for each pixel which is being examined for 
depth, where the cost value comprises respective measures of a number and extent of 
transitions in luminance and/or color and/or color components for pixels of the image on 
a path related to the spatial disposition of objects of the image, wherein said computing 
includes computing a cost value for a first one of the pixels of the image by 
accumulating differences between luminance and/or color and/or color component 
values of pairs of neighboring connected pixels at transitions which are disposed on a 
path from the first one of the pixels to a second one of the pixels, wherein the second 
one of the pixels belongs to a predetermined subset of the pixels of the image, and 
assigning a depth value corresponding to the first one of the pixels on basis of the 
calculated cost value. AAPA however teaches that is known in the art to provide 
algorithms that result in relative depth orderings, and further, to supply depth values as 
a first derivative of the depth value. The examiner, therefore, considers obvious that, 
since the depth determination disclosed in Zheng is a relative depth between the two 
segments delimitated by the contour, that is, the change in depth across two segments, 
one of ordinary skill in the art would have understood that Zheng provides a qualitative 
depth value that comprises the first order derivative of the depth as taught by AAPA. 
One of ordinary skill in the art at the time of the invention would therefore find it obvious 



Application/Control Number: 10/597,976 Page 7 

Art Unit: 2628 

to obtain global depth information for the pixels in the image provided with segment and 
relative depth information as taught by Wilinski and Zheng through the integration of the 
first order derivative of depth provided. In order to do this, the Examiner considers that it 
would have been obvious to one of ordinary skill in the art at the time of the invention to 
examine the different transitions along a path from the pixel to one of the edges of the 
image and to sum these differences which translate in depth differences as taught by 
Zheng. Reconstructing shape information from shading through the use of paths is well 
known in the art, as shown, for example in Wu (see abstract). Wu further teaches that 
why "we take multiple paths in equation 2.6 is based on that the error in depth 
estimation can be reduced by averaging. For the sake of simplicity, we usually choose 
two paths)" (see par. immediately below equation 2.7). The examiner further points to 
equation 2.6 and 2.1 for integrating the estimated depth variations with respect to the 
spatial change; and formula 2.8 an immediately surrounding text for calculating the 
depth at a pixel in the discrete domain using the sampling spatial intervals as well as the 
change in depth regarding space, as well as the summation term that takes into account 
the number of pixel transitions between values. The examiner further notices that there 
is nothing particularly novel in the use of the formula 2.8, which is merely a known 
expansion of formula 2.6. The Examiner also further notes that the change in depth 
between the pixels along the path is obtained independently from the method disclosed 
in Wu, by a shape from shading algorithm, see section III., first par., and further, that the 
change in depth over space disclosed in Wu, since depth is obtained from shape from 
shading indeed constitutes a change in luminance over space, as taught by Zheng. See 
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also p. 3, lines 1 0-1 6 in the Applicant's specification for the equivalence of accumulation 
and integration. Using a line integral such as that taught in Wu would have been 
obvious to one of ordinary skill in the art in view of Zheng, Wilinski and AAPA, since as 
it is well-known in the art, the integral is the reverse of the derivative, and therefore, the 
line integral across boundaries that provide a slope of depth naturally provides the total 
depth difference between the beginning of the line and the end, again as is well-known 
in the art. 

Since Wilinski explicitly teaches both segmenting an image and estimating the 
depth of the pixels by known methods, as discussed above; Zheng explicitly teaches 
shape from shading to detect relative depth between segments, and explicitly suggests 
previously performing image segmentation, as also discussed above, and finally Wu 
teaches using relative depths obtained from shape from shading in order to assign 
global depth to pixels in an image, and since Zheng explicitly suggests using 
segmentation and relative depths between segments, the Examiner considers that it 
would have been obvious to one of ordinary skill in the art at the time of the invention to 
use a shape from shading method such as that taught in Zheng as the depth generating 
method to be used with Wilinski. Further, because Wu teaches a method of obtaining 
absolute depth from relative depth information obtained from shape from shading, the 
Examiner considers that it would have been obvious to one of ordinary skill in the art at 
the time of the invention interested in obtaining the absolute depth of all the segments 
obtained by the combination of Wilinski and Zheng to use the method disclosed by Wu, 
using the relative depth provided by the combination of Wilinski and Zheng. Such a 
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combination would obtain the absolute depth of the objects in the image from the path 
integrals disclosed by Wu applied to the edge pixels at the transitions between objects, 
as discussed by Wilinski and Zheng. 

Regarding claim 2, Wilinski, Zheng, AAPA and Wu do not explicitly teach that the 
predetermined subset comprises one selected from the group consisting of (i) pixels 
which are located at a border of the image, (ii) pixels of a part of the border, and (iii) a 
central pixel of the image". Wu however teaches setting the reference pixel at the center 
of the image, see p. 593, Implementation subsection, 3 rd par.; where the reference point 
is used as the target to create paths and is given and assumed depth, see p. 592, par. 
below formula 2.6. The Examiner considers obvious that when the image is assumed to 
have the object in the foreground as the central object in the image, which is a 
reasonable assumption for most images, the pixels at the edges of the image will more 
likely be the background and the pixel at the center will more likely be the foreground. 
Under that assumption, creating a path from the pixel being measured to the edge of 
the image would give a good approximation of height with respect to the background, 
and creating the path to the center of the image would give a good approximation to the 
foreground. Therefore, it would have been obvious to one of ordinary skill in the art to 
use or combine such paths in order to obtain a global depth value for the pixel, when no 
previous knowledge of depth is available. 

Regarding claim 4, Wilinski, Zheng, AAPA and Wu do not explicitly teach that a 
second one of the differences is equal to an absolute value of difference between 
respective values of neighboring pixels which are disposed on the path. Wilinski 
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teaches that the difference is compared to a threshold; see p. 684, 2 nd col., 3 rd par., and 
a depth value is only computed if the difference is above that threshold. Obviously, 
when applying a threshold to the difference in intensity value, the absolute value would 
be used, since the difference value could be either negative or positive. 

Regarding claim 7, Wilinski, Zheng, AAPA and Wu do not explicitly teach that 
"the cost value for the first one of the pixels is computed by accumulating the 
differences between the values of the pixels which are disposed on the path", although 
the examiner considers this limitation obvious as discussed. Zheng further teaches that 
the pixel values are adjusted to meet a predetermined threshold, see discussion for 
claim 4 above. 

Regarding claim 8, Wilinski, Zheng, AAPA and Wu do not teach that the cost 
value for the first one of pixels is computed by accumulating products of differences 
between the values of the pixels which are disposed on the path and respective 
weighting factors for the differences. The Examiner however considers that, since 
Wilinski teaches that the pixels belonging to a segment receive the values for that 
segment, as discussed above, and the image is segmented in order to distinguish 
contours delimiting surfaces of the objects, and as taught by Zheng, determining the 
values of the edge pixels for the edge pixels requires local information, which could 
diffuse the value of depth for the transition at the edge where the pixel belongs (see p. 
684, 2 nd col., 4 th par.) one of ordinary skill in the art at the time of the invention would 
have found it obvious to accentuate the differences between adjoining uniform surfaces 
by weighting the differences so that the objects silhouettes would have been more 
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clearly defined, for example by weighting more heavily those differences that are closer 
to the pixel measured, thereby increasing the contrast of the resulting depth image and 
compensating for the possible sampling error incurred in the sampling of neighboring 
pixels for edge pixels. 

Regarding claim 9, Wilinski, Zheng, AAPA and Wu would have rendered obvious 
to one of ordinary skill in the art at the time of the invention that a first one of the 
weighting factors which is related to a difference between a value of a particular pixel 
and a value of its neighboring pixel, is based on a distance between the particular pixel 
and the first one of the pixels (see discussion for claim 8, above). 

Regarding claim 10, Wilinski, Zheng, AAPA and Wu would have rendered 
obvious to one of ordinary skill in the art at the time of the invention that a second one of 
the weighting factors which is related to a difference between a value of a particular 
pixel and a value of its neighboring pixel, is based on the location of the neighboring 
pixel related to the particular pixel (see discussion for claim 8, above). 

Regarding claim 1 1 , Wilinski, Zheng, AAPA and Wu implicitly teach computing a 
second cost value for the first one of the pixels of the image by accumulating 
differences between luminance and/or color and/or color component values of pixels 
which are disposed on a second path from the first one of the pixels to a third one of the 
pixels which belongs to the predetermined subset of the pixels of the image (see Wu, p. 
592, 1 st col., par. below formula 2.7 for using multiple paths for reducing errors in 
measurement, see rejection for claim 1 ). 
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Although Wu teaches using the average of the relative depths obtained along the 
path to resolve conflicting cost values for the same pixel, the examiner considers 
obvious that where two or more conflicting measurements for a unique value are found, 
there are a limited number of choices on how to solve said conflict. Therefore it would 
have been obvious for one of ordinary skill in the art to modify the method disclosed in 
Wu, with a choice of value when two or more conflicting values are found for the same 
measurement to resolve this conflict by either choosing the larger value, the lower 
value, the average value or some weighted combination of both as also taught by Wu in 
the cited section. Arriving at the conclusion that the minimum value is more likely to be 
correct would have come naturally to one of ordinary skill in the art at the time of the 
invention after normal testing of the method, if, for example, it was found that the most 
frequent error in measurement is to overestimate the value for a point. 

Regarding claim 12, Wilinski, Zheng, AAPA and Wu also teach computing a 
second cost value for a third one of the pixels on basis of the cost value for the first one 
of the pixels (in Wilinski, see p. 10, lines 26-28 for assigning each pixel the pixel value 
of the segment it belongs to and discussion above for calculating the depth of the pixels 
in the contour in order to get that depth). 

Regarding claim 13, Wilinski, Zheng, AAPA and Wu also teach "computing the 
second cost value by combining the cost value of the first one of the pixels with a 
difference between further values of further pixels which are disposed on a second path 
from the third one of the pixels to the first one of the pixels" (see discussion for claim 1 1 , 
above, for how Wu teaches averaging the results obtained from two paths). 
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1 . Claim 5 is rejected under 35 U.S.C. 103(a) as being unpatentable over Wilinski et 
al. (WIPO Publication No. 02/095680; "Wilinski" hereinafter) in view of Zheng et al. (Q. 
Zheng, R. Chellappa; Estimation of llluminant Direction, Albedo, and Shape from 
Shading; IEEE Transactions of Pattern Analysis and Machine Intelligence, Vol. 13, July 
1991; "Zheng" hereinafter), Redert et al. (WIPO Publication No. 2004/066212; "AAPA" 
hereinafter) and Wu et al. (Z. Wu, L. Li; A Line-Integration Based Method for Depth 
Recovery from Surface Normals; IEEE, November 1988; "Wu", hereinafter) as applied 
to claim 1 above, and further in view of Cahill et al. (U.S. Patent Publication No. 
2004/0062439, "Cahill" hereinafter). 

Regarding claim 5, Wilinski, Zheng, AAPA and Wu also teach that "the values of 
pixels correspond to one of luminance (....)" (see discussion for claim 1, above). 
Wilinski, Zheng and Wu do not teach that the values of the pixels are measured in terms 
of color. Cahill however also teaches a method of generating a depth map where color 
is used in conjunction with luminance for segmentation (see par. [0002]). 

Because Wilinski, Zheng, AAPA, Wu and Cahill teach methods of creating depth 
maps from a 2-D image, it would have been obvious to one of ordinary skill in the art at 
the time of the invention to modify the pixel evaluation of luminance in Wilinski, Zheng 
and Wu with the alternative evaluation of chrominance taught in Cahill because both 
techniques are interchangeable and combinable to give a more precise result, one 
applying to grey-level pictures, and the other to color pictures, and both are well-known 
in the art as taught by Cahill. 
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2. Claim 14 is rejected under 35 U.S.C. 103(a) as being unpatentable over Wilinski 
et al. (WIPO Publication No. 02/095680; "Wilinski" hereinafter) in view of Zheng et al. 
(Q. Zheng, R. Chellappa; Estimation of llluminant Direction, Albedo, and Shape from 
Shading; IEEE Transactions of Pattern Analysis and Machine Intelligence, Vol. 13, July 
1991; "Zheng" hereinafter), Redert et al. (WIPO Publication No. 2004/066212; "AAPA" 
hereinafter) and Wu et al. (Z. Wu, L. Li; A Line-Integration Based Method for Depth 
Recovery from Surface Normals; IEEE, November 1988; "Wu", hereinafter) as applied 
to claim 12 above, and further in view of Nakatsuna et al. (U.S. Patent Publication No. 
2002/0154116). 

Regarding claim 14, Wilinski, Zheng, AAPA and Wu further teach that "cost 
values corresponding to respective pixels of the image are successively computed on 
basis of further cost values being computed for further pixels" (see discussion for claim 
1 , above). Wilinski, Zheng and Wu do not teach that a first scan direction of successive 
computations of cost values for a first row of pixels of the image is opposite to a second 
scan direction of successive computations of cost values for a second row of pixels of 
the image, although Wu teaches performing the computations on a row-by-row basis 
(see p. 593, subsection Implementation, first par.). Nakatsuna, however, teaches a 
method of interpolating depth values on a pixel-by-pixel basis (see par. [01 19] and 
[0165]), in which the pixels are evaluated in a zigzag path, so that pixels may be 
positioned in a two-dimensional neighborhood (see par. [0179]). 

Because Wilinski, Zheng, AAPA, Wu and Nakatsuna disclose evaluating depth 
for pixels on a row-by-row basis, it would have been obvious to one of ordinary skill in 
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the art at the time of the invention to combine the depth measurement method as 
disclosed in Wilinski, Zheng, Wu, with the zigzag inspection path disclosed in 
Nakatsuna. Such an approach would be representative of the well known principle of 
locality in program optimization, by which it is advantageous to perform tasks in such an 
order that the values that have just been calculated and are therefore readily available, 
are those needed to perform the next calculation. 

Conclusion 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Carlos Perromat whose telephone number is (571) 270- 
7174. The examiner can normally be reached on M-TH 8:30 am- 5:00 pm EST. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Kee M. Tung can be reached on (571)272-7794. The fax phone number for 
the organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a 
USPTO Customer Service Representative or access to the automated information 
system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 
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